[ENH] BEP047 - Add audio/video recordings to behavioral experiments - #2231
[ENH] BEP047 - Add audio/video recordings to behavioral experiments#2231bendichter wants to merge 73 commits into
Conversation
Add comprehensive support for audio and video recordings in behavioral experiments: - Add audio file extensions (mp3, wav) and video file extensions (mp4, mkv, avi) with corresponding _audio and _video suffixes - Document usage of audio/video recordings in beh directory for capturing vocalizations, speech, facial expressions, and body movements - Add metadata schema for audio/video device information and stream properties - Include privacy warnings about personally identifiable information in human subject recordings - Update behavioral experiments title to remove "with no neural recordings" restriction, clarifying data can be stored with or without neural recordings - Add examples for file organization including multi-angle recordings and split files - Define optional entities: task, acquisition, run, recording, split
yarikoptic
changed the title
yarikoptic
added
the
schema
…ee macros - Change section title from 'Behavioral experiments' to 'Behavioral recordings' - Convert file tree examples to use MACROS___make_filetree_example for consistent rendering - Address review comments from @yarikoptic in PR #2231
effigies
left a comment
effigies
left a comment
There was a problem hiding this comment.
Overall this makes sense to me. It would be good to get some feedback from contributors to related BEPs, such as eye-tracking (20), motion (29), stimuli (44) and physio (45). Even if this PR doesn't propose adding this as an associated file to those data types, the potential is there and it's worth getting opinions and identifying potential conflicts.
cc @bids-standard/bep029 @bids-standard/bep044
cc @mszinte @julia-pfarr @oesteban (BEP020)
cc @m-miedema @smoia @SouravKulkarni (?) (BEP045)
effigies
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2231 +/- ##
=======================================
Coverage 83.09% 83.09%
=======================================
Files 22 22
Lines 1698 1698
=======================================
Hits 1411 1411
Misses 287 287 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-authored-by: Chris Markiewicz <markiewicz@stanford.edu>
|
A great usecase came from @xiaonansun is video recording of epileptic patients along with EEG or iEEG or other (single units) recording. |
|
and @mvdm 's lab records rodent faces for eye tracking + potential whiskers analysis etc! |
|
I wanted to ask whether is considering support for subtitle or caption files, such as .srt or .vtt, associated with the recordings. In our MoBI arts datasets, especially a Balinese performance project, subtitles could be useful for storing time-aligned transcriptions in the original language, as well as English translations. Would subtitle files be better represented as external files linked to the video, as _events.tsv annotations, or should subtitle/caption formats be considered as part of the audio/video extension? |
Per the July 1 BEP047 discussion, generalize the camera-position field so it also covers audio recording devices (for example, microphone arrays): - Rename the `CameraPosition` metadata object to `DevicePosition` and broaden its description to any recording device. - Move it into the `AudioVideoImageDevice` sidecar group so it applies to the `audio`, `video`, `audiovideo`, and `image` suffixes, and drop the separate `CameraProperties` group. - Update the behavioral-experiments tables and example sidecar accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # src/schema/objects/extensions.yaml
9418f93 to
07ca171
Compare
The BEP047 metadata field was generalized from `CameraPosition` to `DevicePosition` (bids-standard/bids-specification#2231) so it applies to any recording device. Update the example sidecars and README accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidation updateThis PR now carries the complete BEP047 proposal in one place. It combines what were previously three separate PRs:
Both upstream PRs have been closed with pointers here, so #2231 is the single canonical PR to review. The branch has been merged onto current What's included (12 files)
|
|
bids-standard/bids-examples#523 is up do date with this branch |
yarikoptic
left a comment
yarikoptic
left a comment
There was a problem hiding this comment.
I think this is in a good shape and would allow also BEP042 to start making use of the metadata fields and macros
|
Since this would relate to the software which would use such recordings, I would love to invite @talmo @MMathisLab @niksirbi @roaldarbol as relating to Please have a look at this PR changes and/or rendered version at https://bids-specification--2231.org.readthedocs.build/en/2231/modality-specific-files/behavioral-experiments.html . It would be great to hear your opinion about the suggested BIDS file naming conventions for audiovideo recordings and metadata to come along in the sidecars. edit: note -- you are welcome to submit an "official" review of the PR |
|
Thanks for tagging me @yarikoptic! I'm quite busy at the moment, but would definitely want to review this. Is there a time horizon / deadline for the review period? |
|
Currently on holidays, I'll have a look in early August. :-) |
|
@roaldarbol checking in about a review |
Each variant now lands as its own session under the beh datatype directory (sub-01/ses-<label>/beh/sub-01_ses-<label>_video.<ext>), following bids-standard/bids-specification#2231 as clip-extractor writes it. A JSON sidecar next to each video carries BEP047's technical keys as ffprobe extracts them plus the demo detail under an encoding-helper key, and a dataset_description.json at the root replaces manifest.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
* Add Inspect-tab demo set generation and EMBER upload scripts scripts/generate-inspect-demos.sh derives 28 short videos from the committed source recording (scripts/data/Video_S1.m4v), each varying one aspect the Inspect tab surfaces: atom layout (faststart, fragmentation), container (MOV/MKV/WebM/AVI), codec and profile (H.264 Baseline/High 10/High 4:4:4, HEVC, AV1, VP8/9, MJPEG, FFV1), GOP and B-frame structure, bitrate behaviour (CBR, starved cap), track properties (rotation, VFR, resolution, no audio) and metadata tags. Each file carries its own explanation in its title/comment tags, and a manifest.json describes the set for future in-app integration. scripts/upload-inspect-demos.sh pushes a generated set to a dandiset on the EMBER archive with the dandi CLI, and the Inspect demos workflow (workflow_dispatch) runs both against EMBER dandiset 000527, keeping the set as a run artifact and supporting a dry run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Limit the demos workflow's GITHUB_TOKEN to contents: read The job only checks out the repo; the EMBER upload uses its own key. Flagged by CodeQL on #27. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Rename the demo set away from the Inspect tab The demos exercise the whole app, not just one tab: the workflow is now "Upload demos to EMBER" (upload-demos.yml), the scripts are generate-demos.sh and upload-demos.sh, and the default asset prefix is demos/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Fix the upload target and lay demos out BEP047-style The dandiset (000527), instance (ember-dandi) and layout are no longer workflow inputs: the workflow now dispatches with no parameters and the demo files land as sub-01/video/sub-01_desc-<label>_video.<ext>, BIDS entity naming with a distinct desc- per variant, the way clip-extractor names its output. The manifest sits beside the videos. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Write the dandiset.yaml instead of downloading it The dandi download of the dandiset record silently produced nothing on the runner, and with the target directory then missing, cp collapsed the sub-01/ level into it. dandi upload only reads the identifier out of dandiset.yaml, so the script now writes that one line itself and copies the demo tree into a directory that is guaranteed to exist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Pass the API key as EMBER_DANDI_API_KEY The dandi CLI reads the key for a given instance from {INSTANCE_NAME}_API_KEY, so for ember-dandi it ignored DANDI_API_KEY and fell back to an interactive prompt, which EOFs in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Include the webm demos and the manifest in the upload dandi upload only considers extensions it recognizes as assets, which silently left the two .webm files and manifest.json behind; DANDI_DEVEL with --allow-any-path uploads the whole tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Made the demo output a BEP047 dataset of one session per variant Each variant now lands as its own session under the beh datatype directory (sub-01/ses-<label>/beh/sub-01_ses-<label>_video.<ext>), following bids-standard/bids-specification#2231 as clip-extractor writes it. A JSON sidecar next to each video carries BEP047's technical keys as ffprobe extracts them plus the demo detail under an encoding-helper key, and a dataset_description.json at the root replaces manifest.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Enriched the sidecars and moved the demo index into dataset_description Sidecars gain ImageBitDepth and, where one can be determined (via ffmpeg's DASH muxer, or built from the level for HEVC), VideoCodecRFC6381; the per-session demo detail moves out of them into dataset_description.json under the encoding-helper key, so one fetch indexes the set. The dataset also now records its CC-BY-4.0 license, the PLOS ONE article the source video is supplementary material of as SourceDatasets, and the generation date in GeneratedBy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Named the source publication in SourceDatasets The entry now carries the article title, authors, year, license and a note that the source recording is its Supporting Information Video S1; the dataset Description says "the original video" instead of naming the reference session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Dropped Version from the source entry and led with Name, Description The publication year moved into the Description; the year is not a dataset version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Published the source recording under sourcedata/rawbids The recording the demos are derived from now travels with them, in the place BIDS keeps a raw original (with its own dataset_description.json, as clip-extractor's rawbids copy has), and the generator fetches it from there when there is no local copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Removed the source recording from the repository The archive now holds it under sourcedata/rawbids/, so the generator fetches it from there and no video blob has to live in the git history. scripts/data/ is ignored, and the workflow passes the archive key to the generate step so the fetch works against a non-public dandiset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Made the source recording ses-original instead of sourcedata The dataset is raw at its outer level, so the unmodified recording needs no sourcedata/rawbids/ layer of its own: it is simply the first session, with the same sidecar every other session gets and an index entry that carries no ffmpeg_args, since no ffmpeg call produced it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Fall back to the old source path for one migration run The archive still holds the recording where it sat before it became ses-original, and a runner has no local copy to publish the new path from. The fallback lets the first run after the move find it, and comes out once that run has published ses-original. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Dropped the migration fallback now that ses-original is published Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn --------- Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
|
@bendichter Finally found some time to look through it. Generally I think it looks very thorough and well thought out. I mentioned in a comment that if the idea with images (or one of the things that it should help with) is regarding reproducibility of pose estimation or other models, then the annotations on those images also need to be stored somewhere. Is that already possible, like is there place those could meaningfully belong? And if so, it may be worth adding that extra bit of context in the section that discusses images and explicitly mentions pose estimation. Great work! I'll be looking more closely at your metadata sidecars for adoption in animovement. |
Such annotations would count as 'derivative' data since they could be tagged or interpreted in multiple ways, possibly even automatically by software BTW this little conveience app exports in BEP047-compliant format (and supports both image type as well as associated text descriptions): https://clip-extractor.brain-bbqs.org/ You can try it out with pose overlay on the SLEAP tutorial data: https://github.com/talmolab/sleap-tutorial-data/tree/main/new_data |
|
@CodyCBakerPhD You could also see https://roald-arboel.com/croppy/ which is also for extracting clips (and command lab video operations). If there's anything I could do to make it better fit BEP047, please let me know on the repo! :-) |
12 participants
fix #1771
edits by @yarikoptic: